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Abstract. The poster discusses the possibilities of technology-assisted peer feedback 
in English as a Foreign Language (EFL) writing classrooms among low proficiency 
students. It is a part of an ongoing research project developed after a study conducted 
in the first half of 2019 (Lazic & Tsuji, 2020a, 2020b). The first goal is to explore the 
effectiveness of in-class activities, which include technology-assisted peer feedback, 
in improving global aspects of writing, i.e. paragraph structure and content, and to 
examine the uptake of peer feedback delivered via an Automated Writing Evaluation 
tool (AWE), Educational Testing Service (ETS) Criterion®. Second, the study looks 
at students’ perceptions. Participants were 15 first-year students taking an academic 


writing class. 
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1. Introduction 


Potentials and drawbacks of peer feedback are well documented (Allen & 
Katayama, 2016; Hyland & Hyland, 2006). In the Japanese context, studies 
explored the use of AWE in writing classes (Koizumi, Asano, & Agawa, 2016; 
Wakabayashi, 2013). In general, less attention was paid to combined AWE and 
peer feedback’s potential in improving writing (Stevenson, 2016). 


To address these issues, we first conducted a study in 2019 to explore students’ 
perceptions about combined AWE and peer feedback (Lazic & Tsuji, 2020a) and 
look at the effects of the engagement with this type of feedback on revision 
uptake (Lazic & Tsuji, 2020b). Based on the recommendations of these two 
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studies and the literature review, the current study asked the following Research 
Questions (RQ). 


RQ1. What is the effectiveness of in-class peer feedback activities 
delivered via AWE in improving global aspects of writing? 


RQ2. What are the students’ perceptions of in-class peer feedback activities 
delivered via AWE? 


2. Method 


Fifteen Japanese EFL learners studying at a Japanese public university participated 
in the study after signing a consent form approved by the university’s ethics 
committee. This writing course aims to teach writing academic paragraphs, 
coherence, and content development. Classes met once a week, and each lesson 
lasted for 90 minutes. 


The study used ETS Criterion® (Burstein, Tetreault, & Madnani, 2013). After the 
initial training on the use of this AWE, students were introduced to one activity 
per class with the purpose of practicing giving feedback while focusing on the 
paragraph’s organization and content (15 minutes). Then, students used ETS 
Criterion® to provide peer feedback on paragraphs written at home (30 minutes). 
Finally, students had to make revisions based on peer and ETS Criterion®’s 
comments (30 minutes). The class ended with writing a short reflection and 
answering students’ questions (15 minutes). The procedure was repeated during 
Weeks 4, 5, and 6 introducing a different activity each week: a textbook peer 
feedback form, group review of one student writing sample (a form designed by 
the instructor), and the use of polite and other useful expressions when giving 
feedback. The instructor gave feedback at the end of the course. 


Students’ reflections were collected after each session and translated into English. 
To discover trends in students’ reflections, we used KH Coder, text-mining software 
(Higuchi, 2016). To detect emerging topics, a co-occurrence network of words was 
used. Writing samples and students’ comments (on three different occasions) were 
downloaded from ETS Criterion®. When analyzing the data, descriptive statistics 
were used (note: due to using Excel, some of the percentages are rounded to the 
nearest percent and might not add up). 
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3. Results and discussion 


To answer RQI, about the effectiveness of peer feedback delivered via AWE, 
we looked at the ETS Criterion® score differences between before/after writings 
(three different writing tasks, Table |), the number and type of comments students 
made during Writing 2 (Table 2), and a writing sample (Figure 1). 


Table 1. ETS Criterion® scores: before and after (first and second attempt) 


Score Writing 1* Writing 2 Writing 3 

Advisory |Before** — After Before After Before After 

l 8% (1) 80% (12) |60%(9) [67% (10) | 47% (7) 

2 23% (3) | 23% (3) | 13%(2) 33% (5) | 20% (3) | 40% (6) 

3 46% (6) 38% (5) 7% (1) 7% (1) 13% (2) | 13% (2) 

4 8% (1) | 23% (3) 

5 15% (2) | 15% (2) 

6 
Note: a *n=13, **percentages (frequencies). As the number of participants is small, the raw number of scores is also 
presented. 


These scores show improvement or lack of it. Writing improved during Writing 2 
and 3, although moderately. In the table, this is a difference between the score 
frequencies across different score categories (from Advisory to 6). For example, 
in Writing 2, for their ‘before’ writing, most students got a score of ‘1’. When 
students rewrote based on peer and ETS Criterion® feedback, the number of scores 
‘1’ decreased, while the number of students who got score ‘2’ increased. 


Table 2. Amount of peer feedback/comments per type of comments (Writing 2) 


Type of comment 

Organization and development (meaning, content, structure) 61% 
Grammar 16% 
Word usage 8% 
Mechanics 5% 
Style 1% 
Praise 1% 
Other 7% 


Note: n=85 (comments) 


During Writing 2, students wrote a paragraph about whether single-sex education 
is obsolete or not, a topic chosen from the ETS Criterion® topic bank. When giving 
feedback on this writing, students commented more on the global aspects of writing 
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than on any other issue. In other studies (e.g. Allen & Katayama, 2016), comments 
on grammatical errors were the bulk of peer feedback. 


Writing samples were qualitatively analyzed to look at the actual changes and 
uptake of comments delivered by peers via an AWE tool. Like the results of score 
analysis (Table 1), this analysis shows that revisions were minimal, and at surface- 
level, e.g. spelling. In terms of adding new ideas and improving the structure 
and the coherence of the paragraphs, students made few changes overall. Due 
to a lack of space and as most students generally made minimal changes, only 
one writing sample is used as representative. Figure | shows where the student 
addressed one surface-level mistake identified by her peer (the mistake and change 
are underlined). Out of 11 comments received, some referred to the content. 
For example, one classmate expressed her surprise about a piece of information 
(Figure 2, a rectangle) but did not suggest any changes. This writing was graded 
as unsuccessful as the student did not support the main idea in the paragraph. The 
low revision uptake and minimal text changes are possibly due to limited revision 
time, insufficient and inadequate peer comments, and low proficiencies (TOEFL 
ITP? scores between 400 and 450). Others found that lower-level proficiency 
and less confident learners comment and revise less (Allen & Katayama, 2016; 
Wakabayashi, 2013). 


Figure 1. Writing samples/Writing 2 before and after: peer feedback and changes 


Attempt 1/Before/Score: 2 
Essay: 


The Importance of Women's Colleges in Japan 

Women's colleges are established to eliminate the distinction between women and men, and even now there are 
many differences between them, so it is important to exist women's colleges. In fact, in the Global Gender Gap Report 
2020, Japan ranks 121st out of 153 countries, and it is the lowest rank so far. There are two main differences. One is 
their income its rank is 108th. According the Basic Survey on Wage Structure 2017, men's income is about 1.5 times 
more than women's income. The other is the number of cabinet ministers its rank is 139th. Actually, there are 20 people 
in cabinet menber, but there are only three women in them now. As | mentioned above, Japanese gender gap is big 
and Women's college is important to eliminate the gap as soon as possible. 


Attempt 2/After/Score: 2 
Essay: 


Women's colleges are established to eliminate the distinction between women and men, and even now there are 
many differences between them, so it is important to exist women's colleges. In fact, in the Global Gender Gap Report 
2020, it shows the gap between women and men all over the world, Japan ranks 121st out of 153 countries, and it is 
the lowest rank so far in Japan. There are two main differences between women and men. One is their income, its rank 
is 108th. According to the Basic Survey on Wage Structure 2017, men's income is about 1.5 times more than women's 
income. The other is the number of cabinet ministers its rank is 139th. Actually, there are 20 people in cabinet member, 
but there are only three women in them now. As | mentioned above, the Japanese gender gap is big and Women's 
college is important to eliminate the gap as soon as possible. 


2. Test of English as a Foreign Language Institutional Testing Program 
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Co-occurrence network analysis of the 46 most frequent words was used to answer 
RQ?2 (Figure 2). The size of a circle indicates the word frequency. Circles of 
different colors are grouped into communities based on their modularity degree 
— a measure used to detect themes in the data (Higuchi, 2016). Pronouns were 
included to understand what students focused on: frequent use of ‘I/my’ indicates 
that students were more interested in using the activity to improve their writing. 
Identified themes are: students reflecting on problems in their writing and how 
the activity made writing better (purple circles); reading helped students to think 
about writing (red); feedback was helpful with rewriting grammar and getting a 
higher score (green); and ability to notice (problems) as their peers pointed them 
out (yellow). 


Figure 2. Students’ reflections (Writings 1, 2, 3); co-occurrence network of 
words, N42, E47, D 0.55 
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4. Conclusions 


Learners engaged in the AWE supported peer feedback activities by providing 
more comments on content and paragraph structure, which is different from the 
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previous studies. Besides, as seen from the reflection analysis, students found the 
activity useful, e.g. by reading their peers’ writing, they understood their writing 
problems better. However, feedback uptake was minimal, as presented in the 
changes in ETS Criterion® scores and qualitative analysis. Although changes were 
minor, compared to the previous study (Lazic & Tsuji, 2020b), students did focus 
on global aspects of writing when commenting. Thus, it may be concluded that the 
combined activity can be used for the student benefit, but with modifications, e.g. 
a longer time for revisions. Consequently, the next research question to answer Is: 
why did learners not make substantial changes to their texts? 
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